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MATHEMATICS ACHIHVEMENT IN SCHOOLS HAVING AND NOT HAVING CMI • 

y 

AN ASSESSMENT OF MICA 
by 

Alan L. Roecks 
Jphn Chapin 
Madison Metropolitan School District 

This paper focuses on the development of Computer Managed Instruction (CMI) 
in the Madison Metropolitan School District in Madison, Wisconsin. The CMI 
effort involved about 900 fourth and fifth grade students in nine elementary 
schools, all having the same mathematics curriculum. During the 1975-76 academic 
year six of the schools used CMI and three of the schools did not have CMI and 
used nuanual systems. Application of Students' t-test on gain scores for fifth 
grade stndonts in CMI and mnnTi^l Fichools reveal no significant difference (p< 
.01) betvjeen groups. 

The remaining program evaluation results for the CMI effort were considerably 
more encouraging, however. These results revealed that the CMI system was 
effective in relieving teachers of cler-Jcal burdens and providing them with 
useful information - thus, facilitating the process of individualization of 
instruction. The findings, furthermore, indicated that equality of instruction 
provided the nine sjii?ols was opparently quite i^at i sf act ory . A final finding 
was that schools underwent few changes in order to implement CMI. 
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The real argument for more educational computing goes 
beyond merely using the computer as an efficient means 
of instruction. It is much more. The computer in the 
schoolhouse is the first essential step towards compu 
ter literacy, the most meaningful, direct link that 
today's students have with the outside world. (p. 46) 

Harold J. Peters, Conputer D ecisions , July 1976 



BACKGROU ND 

Wiley (1975) points out the declining rate of productivity in the 
economic sector of the United States has been receiving considerable 
attention during the past decade. However, concern about educational 
productivity has been expressed only recently, despite the fact djiit 8 
percent of the American gross national product has been earmarl^cd f or 
education. Rochart and Morton (1975) have stated the use of computers 
in education has increased at an annual rate of about 20 percelr^since 
1971. Lippey (1973) suggests three classifications of computer appli- 
cations in education, indicating computers are used as research tools, 
for student programs, and for operational support. Among those appli- 
cations of the computer in support of instruction, he contends, there 
appears to be an increasing interest in using the computer to assist 
in managing instructional activities rather than administering instruction. 

Nearly all such Computer Managed Instruction (C.M.I.) activities 
support soiuo. aspect of individual i>:at:Icin of ins rructi on . A CM] 
system should have as its primary objectives collecting and processing 
information on students (Spuck, Hunter, Owen and Belt, 1975). This 
information is then disseminated to educational decision-makers so 
that it will have direct and forceful impact on the process of instruc- 
tional decision making. Cooley and Glaser (1968) in describing the 
role of the computer stated: "The function of the computer in a CMI 
system focuses upon allowing better information flow to the complicated 
decision process on a continual basis." (p. 1). At the operational 
level, Bolton and Clark (1973) maintain a dual function exists for a 
CMI System: to utilize the computer to optimize the learning environ- 
ment for each child and to maximize the efficient use of human and 
material resources. Chapin and Roecks (1975) discuss some of the 
issues involved in implementing a CMI system at the classroom level: 

Teachers in individualized programs face diversified student 
populations whose needs are only truly met with an instructional 
system that can cope w^ith students w^ho have highly varying 
basic abilities, educational experiences, and basic skills. 
These students, whose rates of learning might vary by as 
much as a factor of six, require individualized goals. 
Uniq'..e pathways through the curriculum, different learning styles, 
individually selected assignments, and alternative assessmenl: 



strategies are required to meet the potential of indi.vidual i- 
zation. "In terms of curriculum materials, the primary problems 
is one of abuiulancG . The last ten years have seen an explosion 
in the amounf and variety of commercial curriculum materials that 
can he used across the subject sptictrum. Just keeping track of 
what is available, much less what is used within a large school 
system, has become a major administrative task. How is a teacher 
to manage the use of mulLi-texts, supplemental materials, learn- 
ing packets, modular units of instruction, multiple tests...? 
All ot these materials musr be inLegr^iLed into the instructional 
system so that at the proper moment the classroom teacher can use 
the correct instructional material to meet an individual's unique 
problem. (p. 7). 

In 1968, the first major CMI system, IMS, became operational 
tl^ro'.r-h the efforts; of the System Development Corporation of Los Angeles 
(Si, Ir-.'-rman, 1968). Since this time, several CMI efforts similar to 
T.M^ liave be-jn develo,>ed. In reviewing these early CMI efforts. 
Baker (1971) has lisr.-d four common functions that were being performed 
by tl;e computer in ea,,h of the CMI systems; test scoring, di'cignosing, 
p: -jsi-:- ibing, and reporting. The major difference between these Oil 
svsLlips, he noted, lay in the relative amount of emphasis actually 
gi.vi'u to each of the four functions. Wiley ( 1975) reports one of 
the :':ost successful CMI efforts has been Kelley's individual ixed 
econ^jraics course v/hich was piloted at the University of Wisconsin in 
1971-72 and is presently being used at Duke University. Cooley and 
Glast.r (1968) describe a matiagenent information systera, IPI/MIS, de- 
signed by the I'ittsburgh Learning Research and Development Center to 
operate a program of Individually Prescribed Instruction. Lcknn 
(i97i) iridicacec the AIMS system provides considerable information 
abou( students and uses this information in the optimization of 
:i:i;;f t-nt:l ion. WIS-S[M, whiirh vms dt-veLoped by rho Wisc oncin l^esoarcji 
and T^e\'. jopmenL Center for Cognitive Learning, supports a program of 
IndividuaUy Guided Instruction for about 1000 elementary students 
(Spuck, Hunter, Owen and Belt, 1975). Two of the more recent CMI 
efforts for which evaluation information is available are Project PLAN 
and MICA. These systems are described in more detail in the following 
pages. 

Project PU\M (Program for Learning in Accordance with Needs) 
which is distributed by Westlnghouse Learning Corporation is a com- 
puter managed individualized learning program that allows each stu- 
dent to learn at his or her own rate in several content areas. The 
computer assists the teacher by planning student instruction, keep- 
ing track of student progress and scoring and analyxing tests. In 
assessing Project PL^N, Patterson and Johnson (1974) found tliat 
eight out of twelve classrooms showed a teaching pattern of relatively 
high individual interaction, low group interaction and high classroom 
management activities. Their research also indicated PLAN teachers 
spet>:l more time in individual and small group instruction than 
regular classroom situations. With regard to student achievement, 
Patterson (1974) ft>und student inaLhematics results from standardized 
tests at grades 4 and 5 to be slightly better for PLAN schools. In 
response to attitude surveys, students in 12 out of 13 PLAN classrooms 
indicated higher satisfaction than did students in control classrooms. 
Teachers in PLAN classrooms indicated higher relative satisfaction 
than did control teachers. 
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MICA (M.Tpagemcnt of Lnstruction v/itli ComputGr Assistance) i.s a 
CMI. softv.'arc system v;hich has been priniari.ly funded by TLrle 111, 
K.S.E.A. l.oc'.^ed in Madison, Wisconsin, this effort has supported an 
indivlduali/.ed mathematics curriculum for the fourth and fifth grade. 
In dev. l .jv'.g M::CA, a group of Madison teachers at Sherman KlcmCMitary 
School n a cooperative effort v;ith the University of Wisconsin 

Laboratory of Hxperimental Design Lo develop a CMI system whose purpose 
was to provide a feasible means of implementing individualization of 
instruction. This effort began in 1972 and after a year of systems 
work MICA was piloted at Sherman School using the teacher-desiiMUid 
math curriculum. 

The development of the MICA softv/are system paralleled further 
development of tlie Sherman math curriculum. Operationally, this meant 
that schools wishing to have CMI subsequently adopted the individuali s:ed 
math curriculum in their schools and actively participated in CMI 
developmental efforts. As of Vail, 1975, nine elementary schools were 
supporting the Madison CMI effort and were using the same curriculum, 
Sherman math. The fourth and fifth grade population involved in this 
effort consisted of about'900 students. 

Duo to limited financial resources, only six of the nine schools; 
were able to receive the full impact of CMI; i.e., only six schools 
had computer terminals in the schools and used the MICA software 
computer package during the 1975-76 academic year. The remaining 
three schools wiio did not have computer terminals and thus, the support 
of MICA, taught mathematics in the same way as the schools having CM] ; 
recordkeeping and other clerical tasks in these schools were performed 
by school staff rather than the computer. These schools were then 
identified as "manual" or "non-CMI" schools. It should be noted that 
i i:*' -f'Tual L;ho:':e o[ \:r.ich i>('\i:K)\r. deceived ''^ • sapp^^i'i -AMt'i t!}.- 
possible exception of Sherman School whose staff provided initial 
leadership — was essentially arbitrary. Those schools rot receiving 
CMI V7er e assured of CMI support \chenever add i t ional f unding became 
aval lable . 



PRO GRAI^l EVALUATION FOR MADISON'S CMI EFFORT 

A rather thorough program evaluation of the Madison CMI effort 
was begun in 1975 and completed over a year later in the Summer of 
1976. This evaluation examined the effectiveness of the MICA software 
system, the quality of instruction provided for the nine schools 
associated vjith the CMI effort, and the nature of change due to imple- 
menting a technologically innovative product (CMI). 

LL iciency of MICA Softw are 

Successful implementation of CMI is dependent on an efficient 
softvjare system which can meet the need^ of the individualized Instruc- 
tional setting- (Baker , 1977). The philosophy involved in the develop- 
ment of such a software system has been articulated in an earlier 
article by Chapin and Roecks (1975): 



Underlying this effort was the development of an acceptaMe 
philosophy of instruction that such a CMI sysnom v/as to support. 
The central principle was that teachorr should be 100% free of 
all clerical or non-instructional tasks and that all data process- 
ing should be done by machines or aides. Teachers should devote 
their time entirely to teaching and they should have all informa- 
tion recpiired for instructional decisions readily at hand. Those 
general feelings were captured in a Madison Public Schools Research 
and Development Department study of the Sherman effort which 
concluded that the real needs of individualization could only be 
met with a utilitarian system that could handle multiple! schools, 
multiple curriculums, and a wide variety ot. instructional settings 
nn:i piiilosophios. Such a system must completely remove the 
clerical burden from the teachers without imposing upon them 
a-Mtrr.ry and unwanted decisions or information. The system must 
bt. JntLUMCtive and fast enough to process inforn:ation at a rate 
i;?c; :il to the speed at which students come to decision points in 
t^. cuiriculum. Neither students nor teachers can afford to wait 
for nroper information. (p. 10). 

With these guidelines in mind, several time and motion studies 
vert' done to ascertain how mucli teaclier time was spent in individualized 
St'l:r r.::^::, with MICA and in settings without MICA support, or manual 
Sottln^.^j. These studies revealed that teachers using the MICA system 
were freed of nearly all clerical burdens previously associated with 
irvli virli.niization; furthermore, these teachers were able to provide 
iH'.n.' in.Ttrucr.ianal time Cor students during math hour (Chapin, Heggland 
an.,! [['irtK, 1976; Roecks, Kurtz and Heggland, 1976). In comparison to 
m-'iv.i il fjettings not having MICA, it was found that tlie individualized 
t\c-.lii\^,s with MICA could handle Information recording and retrieval 
t::!".;>s faster with less labor required and that lei^s student time was 
s-.^=-it in waiting activity (Chapin, 1976; Chapin, Heggland and Kurtz, 
l?v'6; Roocks, Kurtz and Heggland, 1976). In summary, these results 
strongly sugg.-:ited that the MICA system had been able to facilitate 
the process of individualization of mathematics instruction. A survey 
of teacher and student attitudes towards the MICA system also support 
this contention (Roecks, 1976 [b] ; Roecks, 1976 [c]). 

Qua lity of Instruction in CMI Schools 

A second emphasis of the program evaluation involved researching 
the quality of instruction provided — a factor which can profoundly 
influence student perf orir mce. The quality of instruction provided to 
stud-^nts is dictated in Large part by two considerations: the effectivo- 
ues.'^ of the instructional c..grnts (teachers) and the overall worth of 
the instructional program (^.rarriculum) . The MICA evaluation staff 
adili^-ossed these two iscsues describing the quality of instruction for 
tht^ f;;.ne schools having Shcinaan math including the six CMI schools. 
It ivMs found that teachers participating in the MICA program not only 
.bciiev».-d and actually had more time available for direct interaction 
wit:.i\ students — ^making them more effective instructional agents (Roecks, 
19/0 [b]). Students in schools having the instructional program 
supporting the MICA system, Sherman math, recorded scores at least as 
hL;;;U as students in schools having a more traditional curriculu:' . 



(Roecks, 1976 [a]). Thus, one factor which strongly influenced stu- 
dent performance — the quantity of instruction provided, was shov/n by 
studies of teacher effectiveness and curriculum worth to be quite 
adequate in the six schools supported by MICA and the three manual 
schools • 



Changes Resulting from Tmpleinenlin^ CMI 

The third emphasis of the program evaluation involved researching 
the nature of change accompanying the implementation of CMI. The 
nature of this problem made this research approach primarily descriptive. 
Three factors were identified as impacting the change. The factors 
included the time at which each school began to use the MICA software 
system, each school's commitment to CMI and the degree of developmental 
prcbisms e>:perif=*.n red by each school. Interaction among and between 
factors was likely present, making the factors somewhat interdependent. 
The effect of each of these factors on the nine schools involved in 
implementing CMI is summarized in Table 1 below. 

Sherman School housed the pilot activities for the Madison CMI 
effort, receiving in the Fall of 1973 the support of a limited version 
of the MICA software package. Sandburg became the sc-ond pilot school 
in 197^f, using an abridged version of MICA. The "official" version of 
CMI was launched in the Fall of 1975 using a completed version of the 
MICA software package. Four elementary schools — Glendale, Muir, 
Schenk and Thoreau — In addition to Sandburg and Sherman participated 
in this effort. The remaining three schools using manua]. systems v;ere 
Emerson, Lapham and Spring Harbor. , 

Tabl^ 1 

Factors Involved in Implementing 

CMI in Nine Schools v/I th Shernmn Nnrli 



CMI Implementation 
Factors 



Elemen- 
tary School 
Having Sherman 
Math 


Used MICA 
Software in School 


Commitment to 
CMI during 1975-76 
Academic Year 


Degree of 

Developmental 

Problems 


Glendale 


Began 1975 


high 


low 


Muir 


Began 1975 


very high 


low-medium 


Sandburg 


Be^an 1974 


med-high 


low-medium 


Schenk 


Began 1975 


high 


low 


Shermaa 


Began 1973 


high 


low 


Thoreau 


Began 1975 


medium 


medium 


Emerson 


manual system 


medium 


very low 


Lapham 


manual system 


very high 


low-medium 


Spring Harbor 


manual system 


med-hlgh 


very low 



Evaluating the cost of the MICA program was straightforward and 
It was demonstrated that the adoption of the MICA program into the 
school nystem would result in a significant cost/student reduction 
over a ten-year period (Chapin, 1976). Attempting to assess the 
effect of CMI (in the form of MICA) on student achievement, however, 
posed a considerably more complex problem. 

The principle problem we faced was a conceptual ore; namely many 
factors influence student performance to a greater extent than CMi~in 
facr CMI^ affects student performance in only an indirect manner. CMI 
directly affects thi\ teacher by providing him or her with useful 
instructional information and relieving the teacher of clerical burdens, 
The instructional milieu is then enriched by a system of CMI. The 
classroom environment, however, is only one of many factors affectinp 
sttident achievement (Harnischf eger and Wiley, 1976). Other potential 
factors which could have influenced performance of students in schools 
participating in the CMI effort included the quantity of instruction 
ir instructional setting, home/community environment, the quality of 
ins- :.\icLionl and social background of teachers and students. 

These conceptual considerations were presented to the Board of 
Education in June of 1976. The arguments were apparently not persua- 
sive and an evaluation of the patterns of achievement for the schools 
involvc^d in the CMI effort was therefore begun in July, 1976. Of 
particular interest v;as research comparing achievement results for the 
siK schools having MICA with results for the three manual schools. 
The remainder of this paper will chronicle the methodology employed in 
this research, the subsequent findings, research limitations and a 
discussion o[ results. 
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As discussed earlier in the paper, research involving the quality of 
education revealed that the quality of instruction was quite adequar.e 
.for the nine .schools participating in the CMI effort. Hence', student 
performance is apparently more profoundly influenced by other factors 
than quantity of instruction. 
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METHODOLOGY 



During the Summer of 1976 a study was beg n lo do research achieve- 
ment outcomes of students in schools part.iclrating in tlie CMI effort. 

The population for this study consisted of all students in the 
nine elementary schools exposed to the Shcr)?»an math curriculum during 
their fifth grade year of 1975-76, ^^S3 in number. The sample con- 
sisted of all fifth grade studenlr, the nine elementary schools 
using Slierman math v;ho had i-e.mained at the same elementary school for 
the past three years, 342 in numbr:r. Thus, 70% of the students were 
present from Grade 3 through Grade 5. The motivation for selecting 
the sample in this manner wa'^ to control for the possible effects due 
to student mobility. 2 An nd^.Mcional benefit afforded by this selec- 
tion procedure was that we could then focus on all students who had 
been exposed to the Sherman Math curriculum for the previous two 
years, since Sherman math (and CMI) is offered during the fourth and 
fifth grade years. 

The a chievement me asures used in this study v;ere student mathema- 
tics results from the Sequentr.al Tests of V.ducational Progress (STEP). 
These tests v/ere administered to students dujring the 1975-76 academic 
year at the beginning and end of their fifth grade experience. Using 
these tests as criteria, we could then examine students performance in 
mathematics during their fifth grade year. The Basic Concepts test 
was administered at both the beginning (September^ 1975) and end (May, 
1976) of the 1975-76 academic year. The Computation test was adminis- 
tered only once — at the end of the fifth grade year. Thus, a period 
of eight months spanned the testing of students entering and leaving 
Grad- 5. 



Resear ch Question I 

Two research questions were addressed by this study. The second 
question is discussed in detail in the next section. The first question 
asked if there were differences between the achievement outcomes for 
students in the six schools having MICA and the three manual schools. 
Stated formally, this question asked; 

QUESTION I : IS THERE ANY DIFFERENCE BETWEEN ACHIEVEMENT OUTCOMES 

FOR STLDF.NTS IN SCHOOLS HAVING MICA C0M1V\IU^.D TO MANUAL 
SCHOOLS? 

Two comparison groups were defined by this question: the 267 
students in the six schools with MICA' and the 75 students in the three 
manual schools. For further information on specific composition by 
school, the reader is referred to Table 1 presented earlier in the 
paper. 



^Student mobiiJty affects some school's achievement scores positively, 
others negatively. In other words, better students move into those 
communities served by some schools, poorer studentn move into coiimui- 
nities served by other schools. 
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Descriptive information for answt^ring this questiou was obtainod 
by computing mean and standard deviation values for both groups. 
These measures ^■: vi detemined for the tv;o Sprin?> assessments (Basic 
Concepts and r . ..^putation tests) as well as the Fall assessment (Basic 
Concept te.': i . 

A mer /i\, gfr-i statistical analysis for student performance nuist 
take intu account both entry (pre) and ending (post) levels of per- 
formance. For this study, a student's "entry" score was his or her 
Grade 5 Fall assessment result; his or her "ending" score was the cor- 
responding Grade 5 Spring assessment result. As the Computation test 
was administered only once in the Spring, results from the Basics 
Concepts test, which was given r.wice, were used. 

A difference score for each student V7as computed by subtracting 
his or her Fall Basic Concepts cest: score from the corresponding 
Spring score. As pxpected for nearly every student this score was 
positive, reflecting a gain in mathematics achJ evement . 

A statistical comparison of the difference scores for both groups 
was made using Students t-statistic (see Hays, 1973). A statistical 
difference (p<.01) bctv/ecn group means indicates achievement differ- 
ences between groups exist. The nature of such differences could then 
be explored by examining the magnitude of differences. 



Resea rch Question II 

The fifst research question romprurcrd ach J evr^ment results for 
students in' schools v;it:h MICA with results from students in manual 
schools. This questiou then was very specific in nature; it wzis posed 
in response to inquiry from the local Board of Education. 

The second research question, on the other hand, was concerned 
with possible differences in achievement results a;.)ong all nine schools 
participating in the CMI effort.* This question then was general in 
scope; it was posed by members of the MICA evaluation staff who wished 
to research the issue of how much an effect (if any at all) that 
implementing a CMI program had on student achievement for partici- 
pating elementary schools. 

The motivation behind carrying out this research was to establish 
whether or not factors previously identfisd as impacting implementation 
of a CMI system actually influenced student performance. (For more 
information on the specific factors, the reader is referred to Table 1.) 
As stated earlier, we had good reason to believe that the implementation 
of a CMI program v/as accomplished without major problems in nearly all 
schools and thus would have little influence on student performance. 
This question then provided a means for empirically validating this 
claim. 
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Stated formally. Research Question 11 is as foil 



'i^HERE ANY DIFFi' RK NCE AMONG SPRING ACHIFA'EMENT PATTERNS 
FOR SCHOOLS INVOLVED IN 1^U'LEMENTING CM! AF'IER ADJUSTMENT 
HAS BEEN M\DE FOR STARTING ACH LEV1':MENT LEVELS? 

In order to address the issues contained in this question properly, 
it was necessary to adjust "ending" or Spring, assessment values accord-' 
ing to the "starting" values in the Fall. The same issue was resolved 
in Question I by computing difference scores for the L^ro groups undur 
consideration. For this question, however, nine groups (each of the 
schools) ^^;ere being conr>idered. ^ 

Two techniques that are often considered appropariate in such 
situations are covariance analysis and blocking (Elashoff, 1969). For 
this study, covariance analysis appeared to be preferable due to the 
difficulty of establic^hing meaningful blocking levels across all nine 
schools. A further discussion of this issue is given by footnote 3, 
presented in the Appendix. 



Cr^i^'i^aJ^ ^s-^^^^'P ^ i^ ns for ANCOVA : 

Before covariate analyses or analysis of covariance (ANCOVA) can 
be applied as a valid statistical technique, certain critical assuiupti. 
must be met. Assumptions critical to the underlying rationale for the 
use of covariance analysis are that assignment to treatments has been 
at random, that the covariate is independent of treatments and that 
there is no treatment - slope interaction (Klashoff, 1969), A discus- 
siorftof how tlie design of this study met these assumptions is given in 
thi.' App'.Mid i 

For now, it is sufficient to say that tv/o of the three crucial 
assumptions are definitely met for this study. The assumptions of 
randomization of students to treatment groups was apparently violated. 
Kirk (1969) points out that ANCOVA can bq useful in situations such as 
addressed by this question provided caution is taken in interpreting 
results. In particular, in such caseo, we can be sure that the 
covariate adjustment has removed all bias. 



A NCOVA Hypo t heses 

ANCOVA was applied to th^^ fifth grade mathematics achievement 
results for tv;o different situations. In each case, the dependent 
measi re was the student's Spring tost score; i.e., for Case 1, the 
dependent measure was the student's results on the Basic Concepts test 
and for Case 2 the student's results on the Computation test. The 
s^ime covariate was used in both cases: namely, the student's score on 
the Fall Assessment of the Basic Concepts test. The independent 
variable under study v.ms the impact of implementing CMI as it effected 
each school. 
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^ i'ocmally we wcro InLorostod in testing the followinr, hypotl 
roseutcd hero in null form: 



HO: CAS[^ J^: There is no> sigaifjcant diff erence aiiiom; adjusLed 
means scores for the nine sc hools (p < ,01) participatim: in the. CMT" 
efTort . ^ The d ependent measu re of inlr^rest is the student''_s scorcT^on 

.^ELiPJi ^^^^^ Basic Concep ts Te^t and the cov ariate' is rl7e* student ' s 
score on the Fall test of Math ' Basic Concepts . ' 

Ho: CASE 2^: There i_s no significant difference amo ng adjusted 
mean scores for the nine schools (p<.01) "participat ing in the CmT 
effort. The dependent measure of interest is ^he studji^nj^'s score on 
the Spring CoBiputa tion test and the covariate is t]n-'~Fitudin\t'li^ 
on tjij^ Fall M ath Basic - Co nce pts lesL. " 

Should the results reveal significant differences amouj; mean 
scores, implcrpenting CMI apparently had an eLTect on .student achieve- 
ment. The nature of this effect could be further explained by employing; 
post hoc comparisons in order to ascertain jusc where differences lie. 



RESULTS 

QUr:STTQN I : IS THERE ANY DIFFERENCE BETWEEN ACHIEVEMENT OUTCOMES 
FOR SCHOOLS HAVING MICA COMPARED TO MANUAL SCHOOLS? 

This cjijostlon was addressed by considering, the achievement tnitcome 
ioL- two p.ruupn: the 267 studt-uts in Vnc six MfCA .schools and lor the 
73 students in the three manual schocls. Mean and standard deviation 
values computed for botli groups on the one Fall measure (Basic Concepts 
test) and two Spring/ measures (Basic Concepts and Computation test) 
are presented in Table 2 below. These results suggest very little 
difference existed between the two groups. 

The results on the Spring assessment of the Computation test 
revonled a mean score of 41.0 for the MICA student group and a slightly 
lower mean score of 40.1 for the manual group. Tlie standard deviation 
values were very similar and were 10.8 and 10.9 for the MICA and 
manual group respectively. 
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TABLE 2 

Descriptive Statistics for Students in Elementary Schools 
Having MICA or Manual Schools 
(Grade 5, 1975-76 Academic Year) 



Standardized 
test 

Nine " 
Schools 
Having 
Shennan 

Math \ 



Schools Having 
MICA (six schools) 



N 
267 



Fall 



STEP test of 
Basic Concepts 



Spri ng 



X S,D, 
!7.3 S.6 



X S.I). 
32. A 9.2 



STEP test of 

Mathematics 

Computation 

Spring 



X S.D. 
AKO 16.8 



Manual Sci:ool« 
(three schools) 



75 



27.1 7.8 



32. A 8.8 



40.1 10.9 



The results for the Basic Concepts test indicated essentially no 
difference between Fall (or starting) mean valuer and Spring; (or 
ending) values. Students in schools havin^^ MICA began Grade 5 with a 
slightly higher mean score (27.3) than students in manual schools 
whose mean score was 27.1. Meavi scores for botii £^roups were identical 
for the Spring testinc; with a value of 32, A beiui^ recorded, 

A t~test based on the difference scores on the Basic Concepts 
test for students in eacl; group revealed no significant differenc.es 
(p<'.OI) l)etv.":pn groups. 

Thus the results strongly suggest there v:as no significant dif- 
ference botv/een achievement outcomes for students in schools having 
MICA and for students in manual schools. 



QUE ST [ON I L: IS THERE ANY DIFFERENCE AMONG SPRING ACHIEVEI^fENT PATTERNS 
FOR SCHOOLS INVOLVED IN IMPLEMENTING CMI AFTER ADJUSTMENT 
HAS BEEN M^VDE FOR STARTING ACHIEVEMENT LEVELS? 

Thii: question was addressed by application of analysis of covaviance, 
wiiere the student's Fall assessment score served as covariate and the 
student's Spring assessment scores were uF-.^.d as dependent varia]}les. 
Two cases were considered, one for each dependent variable. For 
Case I tlie dependent variable was the student's Spring Basic Concepts 
test score; for Case 2, the dependent variable v/as the student's 
Spring ConipuLation test score. 

Tlie disappointing; re.sults revealed F-statLstics wiiose value was 
less than I. The F-statistics reflecting the degree of difference 
among mean scores for Case 1 v;as .86 and .97 for Case 2. The ANCOVA 
tables for Cases 1 and 2 are presented in Tables 3 and A as shov/n 
bcl'.n;. Mc!an and standard duviatlon values for each school on all 
three tusts are given in the Appendix. 
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F-rntios less than 1 generally indicatie that the statistical 
analysis is suspect. Hays (1973) suggests that such a condition 
results when randomization has not h^cn successful. Thus, it appears 
that Question II cannot be addressed employing the technique oC analysis 
of covariance. 

Table 3: A^ICOVA Table for Case I 

Dependent measure: STFiP Test of Mathematics Basic Concepts, 

Spring Assessment 

Covnriate: STEP Test of Mathematics Basic Concepts, Fall Assessment 

AMCOVA Table 



Sou cc.ii 


IX^grees of KroGclom 


S, 


.J,^ 





S. 


F 


Amony SchooLs 


8 


139. 


.71 


17. 


A 6 


.86 


Error 


332 


670A. 


.52 


20. 


19 




Kqual. Ley of S.l.opeb 


8 


115. 


.8 


19. 


A 9 


.96 


Kr ror 


3 2 A 


6548. 


,63 


20. 


21 





T.-ih.lc A: AN-COVA Table Lui: Cas.: v 

iX^pcndent measure: STHF Test ol Mathematics Compu La ti iiU , 

Spring Assessment 

Covariatu: STEP Test of Mathematics Basic Concepts, Fall Assessment 

AK-COVA Table 



Source; 


Dct^rt'f.^, of rrcedom 


S.S. 


M. S. 


Y_ 


Among Schools 


8 


372. 16 


A6.5 


.97 


Error 


332 


15991 .24 


A8. 17 




Kc|iialLLy of Slop.is 


8 


279.27 


3/.. 91 


.72 


Error 


32A 


1571 1 .98 


A8. A9 





liTMLTATlOXS AND TMPLTCATIONS 



Listed below arc some of the limltatLons of this study v/hlch tho 
render should keep in mind v;hile reviewing', the findings ])i-osented. 
Tho first set of limitations are spncifie to Research Question 1, 
v/hLch examined differences between achievement results for students in 
schools with MICA and for students in manual schools. 

1. The purp,c?;sp of Computer Managed Instruction (CM!) is to facilitate 
individualization of instruction in r, continuous progress setting 
by relieving the teacherj--' clerical, i-urden and freeing up more of 
their time to directly interact with students. The computer, 
when used in this manner, can then be seen as directly nffe.ctinp, 
the teacher, but only indirectly affecting the stuTTent. This 
study examined the possible effect of a CMI project (MICA) on 
student achievement. It is unclear just he ^ conclusive the 

. results of such a study can be. 

2. Educational innovations such as CMI often take several years to 
be assimilated into or rejected by the educational community. It 
might very well be that th^^ full impact of a CMI project such as 
MICA has yet to be realized. 

3. The Sherman Math curriculum (and CMI) was used for students in 
both the fourth and fifth grades. This study dealt with only tiie 
impact of the curriculum (and CMI) for students during their 
fifth grade experience. Future studies should research perfor- 
mance at the fourth grade level. 

would include the attitude and ability oi the teaching staff, the 
overall climate of the school, and the environment of the neighbor- 
hood in which the school is located. 

3. Difference scores, v:hich are notably unreliable (Lord and Novick, 
1969), were used to compare achievement results between students 
in MICA schools and manual schools using a t-statistic. This 
procedure, however, appeared to be the best procedure available 
for ansv/ering the question posed. 

6. The results of this study are only generali?.able to students who 

have received their education in the Madison schools while remaining 
in the sane elementary school. Mobility patterns vary considerably 
among schools, depending primarily on the characteristics of tlie 
community in which the school is located. 

The foilov;lng limitations and implications are relevant to Research 
Ques tion I f . This question was not properly answered because the 
results of the covariance analysis indicated the procedure was not 
appropriate for the data under consideration. A forewarning of this 
severe limitation was given when t'ne critical assumptions underlying 
ANCOVA were examined. 
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The use of intnct groups croatics intcrprc tci t Ion di Cf icul. t:.i os that, 
are not present wlicn the random assignment is used in formino the 
experimental groups. This was the case for the results of Quest Lon Jl, 
v/licre the recorded F-ratlos from ANCOVA results were all less than. 1. 
Hays (1973) su^^p,ests that r-r:itios whose values are less than I often, 
result v/hen randomization lias not been successfully accL»niplished . 
Further disrussing this limitatioa Kirk (1969) st.ites: 

Even when analysis of covariance is skillfully used, we can never 

be certain tliat some variable that lias been overlc-oked will not 

bias the evaluation of tlie experiment. This problem is abse[it in 

properly randomized experiments because the effects of all uncontrolled 

variables are distributed among groups in such a vjay they can be 

taken into account in tlie test of significance. The use of 

intact groups removes tliis safeguard. p. /»56. 

AL^ ^T'n a t: ive Approaches 

Given that ANCOVA is not approprLate for addressing this tjuestion, 
one may ash if there is another statistical technique that is appropriate. 
In response, several otlier teclinique:. v;ere considered including; blocking', 
and the analysis of variance (ANOVA) of difference scores. Tlu*se 
tech[iiques, like ANCOVA, appear to h.ivo inherent v^/caknes3es h.owover. 

Establishing meaningful l>locking levels proved to be administratively 
i.Mpossible dua to the differing ranges in test score values recorded 
for cacli of the nine schools. 

A limitation stated for Question I was that difference scores 
tend to be unreliable (Lord and Novick, 1968). This limitation is 
much ::iore profound and far-reachin- for Research. Question II— inakLng 
C-cnr idencc in Lhi.* aLCur-.j^'y of ri-sa.") i s very rIouht.".rul if nhs; -'M^-rit 
analyses were performed using differcsice scores. 

An alternative and less sopliLs tica ted approach involves applying 
a one-way ANCOVA ro each of the three achievement measures recorded 
for each. This analysis would then provide information specific to 
each assessment of eacli test. Subsequrjot application of ANCOVA to 
Spring assessment results revealed significant differences (p<.Oi) 
existed among schools on both the Basic Concepts and Computation 
tests. Statistical signficance was not attained, hov;ever, when ANCOV^A 
was applied to Fall testing scores on the Basic Concepts test. 

Some readers may question our rationale for including the research 
involved witli Question II — especially in light of the fact that tlie 
question vv^as never really ansxN^ered. It is our contention as re ^earcliers , 
hov7ever, that v/e learn from our failures as v^eil as our successes. 
Other researchers, for example, could have encountered similar cir- 
cumstances and resolved tliem in an alternative manner. Our approacli 
could tlien stimulate an alternative direction to be follov/ed. Or 
perhaps our conceptual approach to the problem will provide insight to 
researchers v.'ho study similar problems in the future. VJc hope tliat in 
sliaring our failures as well as our successes v;e can contribute to th(^ 
accumulat in^t» body of knov;ledge in educational research. 



This paper presented a brief history of the development: of CompuLer 
Mnna^ed Instruction (CMT). The focus of this paper has been on the 
development and subsequent evaluation of the CMl effort located in the 
Madison Metropolitan School District in Madison, Wisconsin. This CMT 
effort included about 900 fourth and fifth grade students in nine 
elementary schools all having the same individualized curriculum, 
Sherman, math. Six of the schools used the computer progra^a supporting', 
the CMI effort, Management of Instruction with Computer Assistance 
(MICA), durinc; the 1975-75 aeadcinic year. The remaining; three schools 
used L\ manual system. 

A sunuTiary of program evaluation results for this CML effort was 
presented. These results revealed that the MICA system was effective 
in relieving the teachers of clerical burdens and providing them witli 
useful information — thus facilitating the process of individuali/.at Ion 
of instruction. The findings furthermore indicated that the quality 
of instruction provided in the nine schools was apparently quite 
satisfactory. A final finding v;as that schools undervcent fev: changes 
in order Lo implement CMf. 

In .lune of 1976 these euconra .;;ing results vr-re prusentod to the 
Madison Board of F.ducation in hopes of receiviiV:^, future funding for 
the CMl effort. Cefore making any commi tmeo t , the Board requested 
that the issues of program cor.t and program effect on stu(h:nt matlu— 
ma tics a o h x e v cnne n t be res e a r c hed . 

As cost evalua.cion of the' MICA program revealed that adoption of 
thij program into the school systein w.juld result in significaiit cost/ 
sLudtmt rcd'.u. t i ons over a ten yc:ar period. 

Attempting to assess the viicci oi CMl oli student achicvument 
posed a prohlem' of considerable conceptual difficulty. Many factors 
influence student achievement to a greater extent than CMl, in fact, 
CMI affects student perfoniiance only in an indirect manner. An effi- 
cient CMI system directly affects the teacher by relieving clerical 
burdens and providing useful in f orr.ati on , therefore enriching the 
instructional milieu. The classroom environment, however, is only one 
of many factors influencing student performance. 

Despite this concc!ptual limitation, it was n^'Cessary to research 
the relat iouc^hip of student achievement and CMI and a study was begun 
in the Summer of 1976. Only fifth grade students v.ere included in 
this study. This study tocnsed on Lv;o questions. The first question 
asked if there were differences in achievement patterns for students 
in scliools v/ith the MICA software in contrast to students in manual 
sf:hools. As expected, the results revealed no significant differences 
betv.'OL'ii g, roLi|)S. 

T\\c purpose of the second c;iiestion v;as to find our if implementing, 
:i CM L p r(v.;rani af f ec ted a school * s student per f orirtance . Tiie fact, that 
studv-ats were not randomly :issigned to schools S'.'vcrely llmltied appli- 
cation o[. coni:nonly used i n f e Tlii t La I statistica.l techniques, resulting 
in the f}uc*stion nnt being ]^ro|<'r]y .iddrcssed. 
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FUTURE DIRECTIONS 



The results for the studies of cost effectiveness and achievement 
outcomes as well as the results for the program evaluation previously 
completed were presented to the Board of Education in August of 1976. 
The purpose of this presentation was to request funding for the second 
semester of the 1976-77 academic year. 

One month later the Board gave their decision which was not to 
become involved in supporting the CMI effort. The primarv reason 
given for this decision vv^as that no appreciable gains in student 
achievement were observed. 

The Madison CMI effort was discontinued vTanuary, 1977 . 



It is beyond the scope oE ttiis paper Lo discuss the many, factors 
influencing this decision or to pose strate<:,i(?s which would have been 
employed to change it. 

The distressing fact is that educational computing in the: form of 
CMI is not present in Madison, Wisconsin, public schools due to lack 
of local funding. Unfortunately, such decisions are becoming alarmingly 
common-place in school districts across the United States. In discussing 
this national trend in a recent article of Co mput er Decis ions , Harold 
Peters laments that, "Despite a rise in performance and a drop in 
prices, computers are being kept out of the classroom by inadequate 
funding,. *• (p. 62), 

If we want our educational system to meet the needs of future 
goTiHT-at Ions — generations v;hich wil l very likely be very dependent on 
computer technology — it is necessary to b.igin employing Khicational 
computing in today-5; claosioonis. As pointed out by Peters (1976) 
e(iucatnrs of today ruisr realize: 

The short-term benefits of increased investment is what is now a 
marginal market will be hard to measure. The long-range impli- 
cations, however, are clear. Investment in educational computing 
today is quite li terall y , an i nvestmen t in our f utur e. (Emphasis 
supplied) (p. 46) . 
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APPENDIX 

1. Crucial assumptions for AxMCOVA 

2. Tables of descr ip tivo statistics 

CRIT ICAL ASSUMPTI ONS FOR ANCOVA 

Several assumptions must be mat for the analysis of covariance to 
be a valid statistical technique. Assumptions crucial to the under- 
lying rationale for the use of covariance analysis are that assignment 
to treatments has been at random, that the covariate is independent of 
treatments and that there is no treatment slope interaction (Elashoff, 
1969). A discussion of how well the design of this study for Research 
Question II met these assumptions is contained in the remainder of 
this Appendix. 

Randomizatio n : 

The analysis of covariance is based on the assumption that individuals 
V7cre assigned randomly and that all groups were treated exactly the 
same except for treatments. Covariance adjustment procedures, however, 
are often recommended for reducing bias due to the covariate in studies 
where the experimenter must work with intact groups. Winer (1971) 
indicates that covariance analysis can indeed be useful where assign- 
ment to groups is not random but must be used with caution— particularly 
if initial biases on the covariate can be controlled directly by use 
of blocking. 

In this study, intact groups were considered as students v;ere not 
randc^inly as'vLr.MccI to schooLs. Trc-;rtiTic-nts in tho form of ench schof>l's 
involvement with CM! appeared to be assigned at random. VJith th.: 
exception of the Sherman Elementary School whose staff provided the 
initial developmental leadership, the exposure of the remaining eight 
schools to CMI was dictated by chance occurrence. ElashofI: (1969) 
states that covariate analyses in such cases should be approached with 
caution , houever , 



Blocking was not us:ed for this study for both practical and methodo- 
logical reasons. Practically, it vjould have been difficult to 
construct homogeneous blocks given the wide range of test score 
distributions of the nine schools.' Methodologically, covariance 
analyses appeared to be preferable in terms of the overall precision 
provided. In comparing the precision of blocking versus covariance. 
Cox, (1957) concluded that if the correlation coefficient P (between 
the covariate and dependent measure) is less than 0.4 blocking is 
preferable, if p is greater than 0.6 covariance is somewhat better 
and if p> 0.8 covariance analysis is appreciably better. In the 
two cases identified for this study r.he ccvrel::tion coefficients 
were 0 . 8A and 0.76 respectively; thus, covariance appeared to be 
preferable to blocking. 
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Thus, it: appecics that the assumpLion of randomization has buen 
violated in terms of raiuloiii assignment of students to schools, 
suiting in intact j^roups (schoolrO heing considered. ANCOVA hovuver, 
cnn bo usefu] in such situations but tlic results must bo interpretecr 
with caution. In particular, we can never be sure that the covarianc 
adjustment has removed all bias; in fact, some bias may still he 
present from a disturbing variable which was overlooked. 



Covar iate is ^t'^^^cpendc2i_L^_o£_' '_e»aLment : 

A basic postulate underlying ANCOVA is that the covaiiatc is 
statistically independent of the treatment effect. To achieve sta- 
tistical independence, the covariate should be measured prior to the 
administration of treatments (which it was) and treatments should be 
assigned to groups at random (as discussed in the preceding section). 
Klashocf (1969) suggests that in analysis of variance of the covariate 
(Fall Btsic Concepts test) may be useful as an indication of whether 
or not crc;atments are affecting the covariate. An analysis of vari- 
ance revealed that there was no significant differences (n<.01) among 
mean c:.:ures for the nine schoo.ls on the Fail Basic Concfpts tei;l . 



'f^'catme nt - Slop e Interact ion : 

The standard covariance aunlyeis procedure rests on the assumption 
that the rr;igressioa of the dependent measure on the covariate v;as 
lineal* and that the slope is the same for all treatment groups. 
Application of a general regtcssioii program revealed that less than a 
Oi\o. vcv 00)1 r i ncre-i's-c: un.'::p.! r ii^ed sun of squares result(»d v;i t h the 
addiciop of a q;: zdratic and/or cubic term. lleace, the assumpi Lou of 
linearity was apparently m.et. 

The test of no-treatment-slope interaction is performed during 
ANCOVA. The results of this test have been formally presented in the 
results section of the pap_ir in Tables 3 and A. For purposes of 
explanation, it is noted here that this assumption was not violated 
for either Case 1 or Case 2. 



In summary, it appears that tv;o crucial assumptions were met: 
namely, that the covariate is independent of the treatment and that 
there was no treatment slope interaction. The third crucial assump- 
tion involving randomization of students to treatment groups v;as 
violatcrd; ANCOVA, hov;ever, can be useful in such instances ]>rovided 
caution is taken in interpreting results. In particular, in such 
cases, we can never ho sure that the covariate adjustment iias removed 
al I bias . 
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Mean and Strr-ndartl Deviation Vnliics on Achievement 
Tosts for Nine Schools Participating in the CMI Effort 
(1975-76 Academic Year, Grade 5) 



Achievement 
Test 



Elemen- 
tary 
School 



STEP Test of Mathematics 
Uasic Conc egjs 



STEP Test of Mathematics 
Computation 



I'all Assessment Spring Assessment Spring Assessment 





N 


X 


S.D. 


X 


S.D. 


X 


S.D. 


Clendaie 


50 


25.6 


9.4 


32.3 


8.2 


38.4 


11.3 


Muir 


A8 


30.6 


8.6 


34.9 


7.4 


45.0 


9.1 


S.indbury 


26 


24 . A 


7,0 


29.7 


6.6 


36.0 


10.3 


Schcnk 


50 


25.5 


7.9 


30.8 


8.2 


39.9 


10.4 


Shermaa 


isl 


2A.9 


7.6 


29.5 


8.2 


38.3 


11.4 


Thoreau 


51 


31.3 


8.1 


35.7 


7.1 


45.6 


8.9 


Emerson 


\h 


27.0 


7. I 


31 .9 


8.3 


40.9 


8.7 


Laiili..! 


J 6 


2}.. 9 




30 . 7 
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Tahle 6 

Pearson Product Moiiiunt Correlation Coefficients 
For Studeiits Remaining in Madison Public Schools 
From Grade 3 through Grade 5 

(N = 342) 



Fail Basic Concepts 
Spring Basic Concepts 



Grade 5 Assessiiient 
(1975-76 Academic Year) 



Spring Hasic 
Conco p t s 



09 



24 



Spring 
Computnti on 

.76 

.80 
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